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Summary and 
conclusions 


The Department of Trade and Industry (the Department) are chiefly 
responsible for implementing Government measures to stimulate 
innovation, defined as the successful exploitation of new ideas and 
new technologies, and increase industrial competitiveness. They do 
this through schemes designed to meet an identified market need. 
The schemes fall into the following two categories: 


e direct financial support by way of grants to companies to 
develop innovative technologies (technology development); and 


e indirect support by way of programmes or activities designed to 
facilitate technology access through the transfer of technology, 
technical skills and best practice, both in management and the 
application of technology (technology access). 


The United Kingdom has a good record in basic research, but a 
relatively poor history of commercially exploiting those new ideas. 
Following the Department’s last major review of innovation in 1993, 
they shifted the emphasis of their support from technology 
development schemes towards technology access activity. The 
percentage of their innovation expenditure devoted to technology 
access is likely to increase from 23 per cent to over 50 per cent in 
the period 1992-93 to 1996-97. 


During 1993-94 the Department reduced the number of staff 
involved in the management of innovation schemes and reorganised 
the remaining innovation staff. In April 1994, the Department’s 
regional offices were amalgamated with those of other Government 
Departments to form integrated regional offices (now called 
Government Offices) and the process began of transferring certain 
innovation services to a nationwide network of 200 Business Links. 


. Against this background of continuing change, the National Audit 
Office examined eight schemes under the general headings of 
technology development and technology access. The schemes 
examined are described in Figure A. At the time of the examination 
in early to mid 1994 the 1993 policy changes and the introduction of 
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Figure A: Summary of schemes examined by the National Audit Office 


Technology LINK A multi-departmental scheme which supports industrially relevant collaborative research between industry 
Development and academia. 


SMART A single small company grant scheme which aims to stimulate innovative and marketable products, run on 
the basis of an annual competition. 
(Small Firms Merit Award for Research and Technology) 


SPUR A single small and medium sized company grant scheme designed to help firms develop new products and 
processes. 
(Support for Products Under Research) 


RIN A single small company grant scheme for firms in designated geographical areas to develop commercially 
viable innovative products. 
(Regional Innovation Grants) 


Technology Biotechnology A scheme which disseminates information to potential industrial users about the commercial potential 
Access Means Business offered by biotechnology. 
Manufacturing A scheme which disseminates information about the application and operational benefits of knowledge 
Intelligence based computer systems. 
Teaching A multi-departmental scheme which facilities the transfer of technology and skills from academia to 
Company Scheme business. 
Managing in A scheme which disseminates information on management best practice, particularly on the need to innovate 
the 90s and manage change effectively. 


Business Links had yet to take substantial effect. And, in May 1995, 
the Department’s 1993 policy shift was re-affirmed in the 
Government’s White Paper “Competitiveness - Forging Ahead”. 


5. The main features of the National Audit Office examination were: 


e A telephone survey of industry to ascertain industry's views of 
the Department’s management of the schemes and industry’s 
awareness of schemes. The survey also assessed the outcome of 
industry’s participation in the schemes. 


e The formation of a working group, including representatives of 
the National Audit Office and the Department, to establish the 
comparative cost effectiveness of schemes. The methodology 
was based on an extension of decision theory which, by creating 
and then weighting a set of performance indicators, allows 
schemes to be evaluated in comparison to each other even when 
they are judged individually by different performance criteria. 
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The method does not offer an absolute measure of cost 
effectiveness but does establish the relative standings of the 
schemes examined. 


Outcome of schemes and relative cost 
effectiveness 


. The main findings on the outcomes of schemes were as follows: 


a) Over half of completed technology development projects have 
been commercially exploited and a third patented. 


b) 60 per cent of participants in technology access schemes found 
them to be suitable for their needs and about one third of 
participants had taken action as a result of participating. 


— 


c) Some 75 per cent of technology development projects were 
“additional” in that the companies perceived that they would not 
have gone ahead without the Department’s financial support. 
Additionality ranged from 95 per cent for LINK to 59 per cent for 


SPUR. 


d 


— 


“Deadweight” is expenditure in excess of the minimum grant that 
companies would have required to have gone ahead with 
projects. For those companies in the survey in receipt of 
Departmental funding under the technology development 
schemes, the National Audit Office survey suggests that the 
Department incurred deadweight amounting to some 

£4.3 million. Estimates of deadweight ranged from 10 per cent of 
total support (RIN) to 53 per cent (LINK). It is not possible to 
eliminate all deadweight; and retrospective assessments by 
companies need to be treated with caution particularly those 
concerned with LINK. 


The results of the comparative cost effectiveness work showed that 
seven of the schemes had similar levels of cost effectiveness but the 
eighth scheme, LINK, ranked significantly lower. This was 
principally due to the scheme’s relatively high administration costs 
and lengthy grant application processing time: the scheme rated 
highly on many aspects of innovation. An analysis of the principal 
factors affecting the ranking of schemes showed that: 


a) although innovation and cost were assigned equal weight, the 
two individual indicators accorded most weight were 
“programme costs” and “running costs”. This militated in favour 


10. 


Ale 


The Department of Trade and Industry’s 
Support for Innovation 


of the cheaper access schemes, as opposed to development 
schemes - and indeed four out of the five most cost-effective 
schemes were access schemes; and 


b) within the group of innovation indicators, those relating to 
projects and products secured a higher weighting than those 
dealing with transfers of information. 


These findings indicate scope to improve performance in two main 
areas: 


e the design and evaluation of schemes; and 
e the promotion and administration of schemes. 


The Department’s procedures in these areas are assessed in the 
following sections, and recommendations made for improvements. 


Scheme design and evaluation 


The Department had in place arrangements to review the overall 
balance of their portfolio of schemes through their annual planning 
round, and to evaluate the results of schemes. These arrangements 
have since been supplemented by measures designed to enhance the 
Department’s ability: to judge cost effectiveness; to identify gaps and 
overlaps in their schemes; and to ensure that a standard core of 
monitoring data and evaluation results are available whilst schemes 
are in progress. 


Scheme design decisions followed the approved processes (Figure B) 
and were generally well supported by information and analysis. In 
two cases examined where amendments to scheme design 
subsequently became necessary the Department had taken the 
necessary action. Objectives were usually set which were both 
commensurate with and, in seven cases, provided adequate or 
better coverage of the scheme’s aims. 


The National Audit Office noted that there was scope for 
improvement in the following areas: 


a) Objectives for technology development were not always readily 
measurable and those for technology access were insufficiently 
focussed on outputs. The Department expect that their revised 
evaluation arrangements and improvements in the scheme 
design process will enhance the quality of their objective and 
target setting. 
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Figure B: Outline of the Department's Scheme 
Design Process 


Demonstrate existence 
of significant 
market failure 


| 


Demonstrate scheme 
“additionality"* 
(scheme must provide benefits 
to eliminate or alleviate the 
market failure) 


| 


Ensure scheme is consistent 
with overall Departmental 
policy objectives 


| 


Set general scheme 
objectives in line 
with rationale 


| 


Set testable 
objectives and targets 


1 


Specifiy project 
appraisal arrangements 


| 


Specify on-going 
monitoring arrangments for 
projects and the scheme 
as a whole 


| 


Specify scheme 
evaluation arrangements 


| 


Specify monitoring 
information available for 
evaluation purposes 


b) Variations in average “deadweight” between the technology 
development schemes indicates the need to look more closely at 
this aspect in the design of, particularly, schemes aimed at more 
complicated projects and medium sized and larger companies. 
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c) Variation in commercially successful outcomes between schemes 
is partly explained by the different nature of schemes. But two 
aspects may have common significance. The technology 
development scheme (SMART) which assesses the quality of 
applications more fully than the other three has the highest 
ratings for successful projects. Access schemes that are designed 
so as to facilitate implementation of new practices or 
technologies, such as the Teaching Company Scheme, rate higher 
than more general awareness schemes. 


d 


— 


Evaluation arrangements, although good and improving, do not 
facilitate a robust comparison of scheme performance, and so it 
can be difficult to learn lessons from the different aspects of 
schemes’ design and operation and assess the most cost effective 
resource allocation. 


The National Audit Office recommend that: 


e innovation scheme objectives should be analysed and reviewed 
as a whole as part of the annual planning and review 
arrangements, on the lines adopted by the National Audit Office 
for this study; 


e the Department should consider adopting more flexible funding 
arrangements for schemes where “deadweight” would 
otherwise be high; 


e the Department should explore the extension of graded 
assessments of the quality of applications from SMART to other 
technology development schemes; 


e the Department should take on board the weight put on 
indicators of the implementation of new ideas and technologies 
in their reviews of the design and targeting of access schemes; 


e the Department should investigate further the development and 
use of the comparative cost effectiveness methodology used in 
this study as part of their evaluation and resource allocation 
processes. 
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Scheme promotion and administration 


13. Figure C provides an outline description of the grant application 
process for technology development schemes and the route to 
participation in technology access schemes. 


Figure C: Outline of route to participation in Departmental innovation 
schemes . 
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Determine individual project 
objectives and targets 


1 


Monitor progress of project 
against objectives 
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Grant paid on the basis of 
progress against objectives 


14. The Department promote awareness of the policy change and their 
schemes in a variety of ways: advertising, mailshots, direct contact 
with industry, via the work of Regional Offices; and, from 1993 on, 
via Business Links. The arrangement for managing the promotion of 
schemes varies in detail from scheme to scheme. In all cases, there 
is considerable delegation to regional and operational staff. 


Loe 
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The National Audit Office found that: 


a) Although there was a relatively high level of awareness amongst 
industry for the SMART scheme, there were low levels of 
awareness for the other seven Departmental schemes, 
particularly those concerned with technology access. 


b) Understanding amongst industry of Departmental policies for 
support for innovation, or of the distinction between technology 
development and access support was low. 


The National Audit Office examined how the Department 
administered the eight selected schemes. They found that, in 
general, schemes were well administered and in particular: 


a) grant applications were usually appraised in line with laid down 
procedures; 


b) eligibility requirements for schemes and the progress of schemes, 
funded projects, activities and events were, in most cases, well 
monitored. 


Areas where there was scope for improvement included: 


a) the low industry awareness of many of the schemes examined, 
and the findings that for two access schemes a significant 
minority of participants were not drawn from the prime target 
audiences, both indicate scope for improvement in the promotion 
of schemes and their delivery to appropriate target audiences. 


b 


— 


the National Audit Office survey shows that most applicants had 
experienced some difficulties with the schemes. The most 
common complaints were of the time taken to process 
applications and of the difficulty and expense of making an 
application. The highest levels of complaint were with the time 
taken to process LINK applications. This finding is supported by a 
recent Office of Science and Technology evaluation - which has 
resulted in proposals to streamline the process. 


c) for most access events examined no participation fee was 
charged. The National Audit Office survey revealed industry 
willingness to pay for such events - a move which could reduce 
costs to the Department and help secure an appropriately 
committed audience. The Department are considering 
appropriate admission fees for such events. 


The Department of Trade and Industry’s 
Support for Innovation 


d) the Department are unable to derive administrative costs for 
individual schemes from their information systems: the costs of 
schemes quoted in this report are estimates prepared by the 
National Audit Office with the Department’s help. The lack of 
good costing information makes it more difficult to manage 
schemes cost effectively. 


18. The National Audit Office recommend that: 


e the Department should include scheme promotion and 
targeting activity in appraisal and review arrangements for all 
schemes; 


e the Department should monitor industry awareness of the 
schemes and the policy change, and track the impact on 
awareness of initiatives such as Business Links; 


e the Department should monitor the success of streamlining 
initiatives for LINK applications, and aim to apply any lessons 
learned more widely; 


e the Department should modify their management information 
systems to provide better information on the costs of 
administering schemes. 


Overall conclusion 


19. Insummary, the National Audit Office found that for the eight 
schemes examined the Department have had reasonable success in 
meeting their objectives, and that they had put in place a sound 
framework of management practices. There is scope, however, to 
build further on these arrangements to improve the overall cost 
effectiveness of the Department’s support for innovation. 
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Part 1: Introduction 


1.1 Industrial innovation can be defined as the successful exploitation of 
new ideas leading to new or improved products, services or ways of 
doing business. Innovation may be speculative or market driven. It 
requires good management and linkages across the range of 
corporate functions - research, design, development, marketing, 
sales, distribution, servicing and administration. It is not 
synonymous with invention although this may be part of the 
innovation process. 


1.2 Whilst United Kingdom industry has a relatively strong track record 
in research, it has proved less successful in terms of commercially 
exploiting new ideas. A ranking of 22 member countries of the 
Organisation for Economic Co-operation and Development carried 
out by IMD Lausanne and the World Economic Forum shows that 
the United Kingdom was ranked 13th overall (Table 1) on the “world 
competitiveness scoreboard”. In terms of the extent to which 
enterprises are managed in an innovative, profitable and 
responsible manner, the United Kingdom was ranked 16th. 


Table 1: World Competitiveness Scoreboard: United Kingdom 
Balance Sheet 1993 


UK 
Position 


Internationalisation - country's participation in international trade investment 
flows. 


Science and Technology - scientific and technological capacity, success of basic 
and applied research. 


Management - extent to which enterprises are managed in an innovative, 
profitable and responsible manner. 


Infrastructure - adequacy of resources and systems to serve the basic needs of 
business. 


People, Education and Skills - availability and qualifications of human resources. 


Overall Average. 


(Rank 1-top, 22-bottom) 
Source: IMD/ World Economic Forum. 


Table 1 shows the United Kingdom's overall ranking of thirteenth in world competitiveness out of 22 
Organisation for Economic Co-operation and Development countries. 
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There are many reasons why firms may not innovate. Funding may 
be difficult to obtain or the required expertise and know-how may 
not be readily accessible. There may also be a reluctance on the 
part of the company to invest in activities that offer no short term 
return, or poor management may mean that successful research is 
not combined with other important features of the innovation 
process thus reducing the chances of successful commercial 
exploitation. Such factors may be particularly critical for small and 
medium sized companies. It may also be difficult for firms to acquire 
newly developed innovative technologies and processes from other 
companies which may be reluctant to share their knowledge with 
potential competitors. These sorts of constraints on industrial 
innovation are the basis for intervention by the Government. 


The Department of Trade and Industry 


The Department of Trade and Industry (the Department) are chiefly 
responsible for implementing Government measures to stimulate 
innovation and increase industrial competitiveness. They employ a 
number of measures designed to achieve these objectives. These can 
be categorised under the general headings of technology 
development, technology transfer and best practice awareness: 


e technology development: a number of schemes exist to provide 
direct support by way of a grant to companies to assist with their 
development of new and innovative technology; 


e technology transfer: indirect support is provided by way of 
schemes designed to facilitate the transfer of technology and 
technical skills between organisations; 


e best practice awareness: indirect support by way of schemes 
designed to disseminate information on best practice in 
management and the use of technology. 


Science, Engineering and Technology White 
Paper 


In May 1993 the Government published their White Paper “Realising 
our Potential - a Strategy for Science, Engineering and Technology” 
(Cm 2250). This reaffirmed the Government’s aim to address the 
market environment that had resulted in the contrast between the 
United Kingdom industry’s excellence in science and technology and 
its relative weakness in exploiting this for commercial advantage. 


ial 


Figure 1: The Department's innovation 
expenditure target 1996-97 


[J Technology Development 
WH Best Practice 
§ Technology Transfer 


Source: The Department, March 1995 


Figure 1 shows the targeted relative size 
of the Department's three main categories 
of innovation expenditure. 


Figure 2: The Department's innovation 
expenditure 1992-93 


2% > 


[9 Technology Development 
@ Best Practice 
HH Technology Transfer 


Overall expenditure £153 million. 
Source: The Department. 


Figure 2 shows the relative size of the 
Department's three main categories of 
expenditure in the year immediately 
preceeding the policy change. 


1.6 
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In parallel with the discussions leading up to the publication of the 
White Paper, the Department undertook a fundamental re-appraisal 
of their support for innovation. The main outcome of this was a shift 
in the balance of support away from generating new technology and 
towards the exploitation and transfer of technology and best 
practice awareness. 


Their revised objective is therefore to stimulate innovation in 
industry by: 


e promoting the importance of innovation and its management; 


e encouraging research and development, the spread of best 
practice and the transfer and diffusion of technology and 
technical skills; 


e encouraging interaction between industry and all other providers 
of technology, including academia, whether based in the United 
Kingdom or elsewhere; and 


e funding research and development. 


The implementation of the shift in policy 
emphasis 


Although the shift in policy emphasis was announced in 1993 it 
could not be implemented immediately. In particular, the high level 
of commitment on existing schemes, some of which were closed to 
new applications from September 1993, means that residual spend 
will continue for a number of years. This means that any change has 
to be made progressively and it will be some time after April 1995 
before expenditure from closed schemes ceases. However, the 
Department anticipate that by 1996-97, technology transfer and best 
practice awareness will account for over 50 per cent of the 
Department’s expenditure directed towards support for innovation 
(Figure 1) against 23 per cent in 1992-93 (Figure 2). The 
Department also re-designated “technology transfer” and “best 
practice awareness” as “technology access”, with effect from 

April 1994. 


The approach towards support for innovation and the funding 


allocations of the European Commission and other developed 
countries are shown in the box opposite. 
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How the European Commission and other developed countries 
encourage innovation: 


e Like the United Kingdom, the European Commission is placing 
greater emphasis on technology transfer although it continues 
to put the greater part of its funding into technology 
development projects. Support is provided both through the 
Commission’s own schemes and by supporting those of 
member states. There is also an emphasis on support for 
smaller companies; and there has been a threefold increase in 
the support for biotechnology, mainly for dissemination. 


e Other developed countries continue to emphasise technology 
development rather than technology transfer or best practice 
awareness. Nevertheless, all countries surveyed by the 
National Audit Office except Australia provide some support 
for technology transfer or best practice. The highest 
percentage of budget devoted to these aspects of support 
outside the United Kingdom is in Germany where the figure is 
22 per cent. 


Staffing and organisational changes 


Figure 3 shows that the administration of the various schemes is 
split between the Department’s headquarters and the regions. Policy 
work is concentrated at headquarters. During 1993-94 the 
Department reduced the number of staff engaged in managing 
innovation at their headquarters by some 90 posts. They also 
implemented a significant re-organisation of the remaining staff 
following the resignation of the Chief Scientist in February 1994. 
Technology functions were distributed amongst the Department’s 
sector divisions. The budget remains centrally managed and a focal 
point for innovation is being maintained at senior management level. 


In April 1994 the Department’s regional offices were amalgamated 
with those of other government departments to form Integrated 
Regional Offices (now called Government Offices). The process was 
also begun of transferring the delivery of certain innovation services 
to industry from regional offices to a nationwide network of 

200 Business Links. In the future these will be a major source of 
information for the business community and some innovation work 
will be based within them. 
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Figure 3: Schemes examined by the National Audit Office 


Description Administered by 


Expenditure (1) 
£Million 


Technology Development Schemes 


A multi-departmental scheme launched in 1986 involving nine Government departments and 
four Research Councils which supports industrially relevant collaborative research between Head quarters(2) 
industry and academia in a number of specified technologies. Available to companies of any 
size. 


A single company grant scheme run on the basis of an annual competition and open to 
firms with less than 50 employees. Support is available in two stages with the emphasis on Regional Offices (3)/ 
feasibility in the first stage and prototype development in the second stage. The scheme Headquarters (4) 

aims to stimulate innovative and marketable products. 


A single company grant scheme designed to help firms with up to 500 employees (250 from 
September 1993) develop new products and processes which involve a_ significant Regional Offices (3) 
technological advance. 


A single company grant scheme available to firms with less than 50 employees in 
designated geographical areas for developing commercially viable innovative products. Regional Offices(3).(5) 


Technology Transfer Schemes 


“ecrintttae lis scheme disseminates information to potential industrial users about the commercial 
Means potential offered by biotechnology. The programme has involved two overlapping phases. Headquarters 
Business The first phase ran for the three years to October 1993 whilst the second phase commenced 
in late 1992. 


This scheme promotes the application of computer software technology of knowledge based 
systems to solve operational problems in: manufacturing technology and management; Headquarters 
product design; marketing; distribution; and customer support and highlights the 
opportunities the technology offered for increasing competitiveness. The programme ran 
for three years to June 1993. 


Manufacturing 
Intelligence 


A multi-departmental scheme launched in 1975 currently managed by a semi-autonomous Teaching 
Teaching body known as the Teaching Company Directorate on behalf of the Department and nine Company 
Company other public sector sponsors. The scheme provides technology access for companies and Directorate 
Scheme facilitates the transfer of technology and skills from academia to business by engaging 
graduates to work for a partnering company for two to three years. 


Best Practice Awareness 


This scheme promotes management best practice in four key areas: production; purchasing; Headquarters/ 
Managing quality and design. It commenced in 1989 following the publication of the White Paper "The Regional Offices 
in the 90s Department of Trade and Industry - The Department for Enterprise" (Cm 278). The scheme 
(Phase 2) is currently in its second three year phase which commenced in 1992. This places a greater 
emphasis on the need to innovate and manage change effectively. 


Notes: 

(1) The Department's expenditure (excluding overheads) in the four years to 31 March 1994. Expenditure by other Government departments 
on LINK and the Teaching Company Scheme is excluded. 

(2) Overall responsibility for LINK passed from the Department to the Office of Science and Technology in late 1993. 

(3) The delivery of certain services currently undertaken by Regional Offices is progressively being transferred to Business Links. 

(4) Headquarters are responsible for approving winning applications under SMART on the basis of appraisals prepared by the Regional Offices. 


49.8 


45.9 


14.8 


20.0 


0.8 


0.9 


21.1 


6.2 


(5) The responsibility for the Department's RIN scheme passed to the Department of the Environment with effect from April 1994 (see paragraph 1.13) 


Source: The Department 


Figure 3 shows the general objectives and target groups of each scheme examined by the National Audit Office, their administrative arrangements and the level 


of spend over the four years to 31 March 1994. 
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Scope of the National Audit Office examination 


Against this background, the National Audit Office examined: 


e industry’s awareness of Departmental innovation schemes (Part 2 
of the report); 


e the framework for and outcome of the Department’s support for 
innovation (Part 3 of the report); 


e performance indicators and the comparative cost effectiveness of 
the Department’s innovation schemes (Part 4 of the report); and 


e the efficiency of the administration of the Department’s 
innovation schemes (Part 5 of the report). 


This report focuses on eight innovation schemes designed to 
promote technology development, technology transfer or best 
practice awareness set out in Figure 3. These schemes were chosen 
because they are representative of the Department’s innovation 
portfolio and, with the exception of the Manufacturing Intelligence 
scheme, which was closed down in June 1993, will continue to be 
funded in the future. LINK and the Teaching Company Scheme are 
multi-departmental but the National Audit Office’s examination was 
concerned only with the parts of these schemes administered by the 
Department. The Department’s Regional Innovation Grants scheme 
(RIN) became the responsibility of the Department of the 
Environment in April 1994. The National Audit Office’s examination 
was concerned with the period prior to this transfer of responsibility. 


The report also concentrates on the support available within the 
eight schemes for two particular technologies: biotechnology and 
advanced manufacturing. This enabled the National Audit Office to 
explore whether there were any significant differences in the impact 
of the Department’s support for a specific technology, such as 
biotechnology or a more broad based technology, such as advanced 
manufacturing. The main features of their examination are set out 
in Figure 4. 


The National Audit Office examination covered the four year period 
to March 1994 and was carried out in the first seven months of 
1994. They acknowledge the willing assistance of all concerned. 


Subsequently, in May 1995, the Government published their White 


Paper “Competitiveness - Forging Ahead” (Cm 2867). This developed 
further the shift in policy emphasis, set out in the 1993 White Paper 
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Figure 4: The main features of the National Audit Office examination 


1: Survey of Industry 


a: The National Audit Office commissioned IFF Research Limited to undertake a telephone survey (the Survey) of industry 
to ascertain: 


@ industry's views on and awareness of the change in policy emphasis and their awareness of the Department's programmes 
and schemes; 


@ industry's view of the Department's management of the programmes and schemes; and 
@ the outcome of industry's participation in the Department's programmes and schemes. 
b: Of the 1,105 companies initially contacted for interview, a statistically representative sample of 660 companies from the 


advanced manufacturing and biotechnology sectors responded to the survey - representing an overall response rate of 73%, 
after taking account of those which did not qualify for interview. 


c: The survey was designed to obtain a statistically valid sample of the views of companies falling into the following three 
categories: 


@ suppliers (or developers) of technology who had applied to the Department for financial assistance under one of four grant 
schemes during the period April 1990 to December 1993 (402 responses): 


@ developers who had not so applied (108 responses): and 


@ companies who were customers (or users) of technology developed by companies of the type described in the two categories 
above (150 responses). 


The telephone survey was complemented by in-depth interviews with 20 companies. These provided evidence to amplify and 
illustrate the statistical findings of the survey. Appendix 1 provides further details of the survey 
2: Comparative Cost Effectiveness 


The National Audit Office established a joint working group with the Department to examine the comparative cost 
effectiveness of the innovation schemes set out in Figure 3. They commissioned Facilitations Limited and an academic 
consultant from the London School of Economics (Professor J. Rosenhead) to assist in this exercise. Appendix 2 gives 
summary details of the methodology and analysis. 

3: Comparisons with Other Countries and Organisations 


The following methodology was used to establish the approach adopted by other Government departments, other developed 
countries and the European Commission: 


a: questionnaires were sent to 11 developed countries of whom nine responded; 
b: National Audit Office staff visited the offices of the European Commission in Brussels and Luxembourg; 


c: Interviews were held with other Government departments. 


4: Case Examinations, Analysis and Interviews 
The National Audit Office: 


a: conducted interviews and the following 48 case examinations at the Department's headquarters in London and in two of their 
Regional Offices - West Midlands and Yorkshire and Humberside: 


@ 18 LINK projects (6 Biotechnology and 12 Advanced Manufacturing) from 9 LINK programmes: and 


@ i “ SMART, 10 SPUR and 7 RIN projects (15 Biotechnology and 15 Advanced Manufacturing) at the two Regional 
ices. 


b: interviewed staff of the Teaching Company Directorate (see Figure 3) and examined 12 Teaching Company Scheme 
programmes; 


c: interviewed staff of the CBI: 


d: examined recent Departmental evaluations of their innovation programmes and schemes; and 


e€: analysed scheme statistics over the period April 1990 to March 1994. 
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(paragraph 1.5), towards support for technology access and 


awareness of the importance of innovation as a key component of 
competitiveness. 
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Part 2: Industry 
awareness of Support 
for Innovation 


This part of the report considers industry’s awareness of the 
Department’s schemes and the shift in policy emphasis. 


Industry’s awareness of the schemes and 
the shift in policy emphasis 


In order to secure improved innovation, the Department must make 
relevant companies aware of the support and events available, as a 
precursor to their participation. In some cases, companies may 
benefit from departmentally-supported research and services 
without being aware of the underlying departmental effort. But for 
the schemes examined by the National Audit Office, the nature of 
support or event required pre-knowledge of the scheme itself. 


Each of the eight schemes examined by the National Audit Office is 
marketed differently. Of the four technology development schemes, 
only SMART is advertised nationally in the press. Neither SPUR nor 
RIN is advertised: awareness of them is normally through contact 
with the regional offices who administer the schemes. Some calls for 
applications for LINK are made in technical journals but otherwise 
applications result from discussions with the Department. 


Mail shots are normally used to contact potential participants for 
technology transfer or best practice awareness events. 


Awareness of schemes and activities is obviously a pre-condition for 
participation. But awareness of the underlying rationale for support 
is also desirable, both to help companies see the value of the support 
offered, and to help them feed their ideas and needs to Government 
in a constructive way. Accordingly, throughout their 1993 review of 
innovation support the Department consulted companies about their 
proposals. And in May 1993 the Department set out to inform 
industry more generally of their decision and its impact on the 
existing schemes. Following a Parliamentary written answer a press 
release was issued explaining the changes and the President of the 
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Board of Trade wrote to a selection of companies. Apart from this, 
the main vehicle for marketing the changes was in the form of 
addenda to existing literature about the schemes, distributed in the 
normal way to potential applicants, and by selective mail shots. 
Regional office staff were provided with briefing packs summarising 
the changes to help them in the process of informing industry. 


To assess whether, by these means, the Department were attracting 
the most relevant participants to their schemes and if industry was 
sufficiently aware of the shift in policy emphasis, the National Audit 
Office examined: 


e whether a sufficient proportion of potential participants were 
aware of the eight schemes covered by this report (See Figure 3); 


e industry’s understanding of the difference between technology 
development, technology transfer and best practice awareness; 
and 


e industry’s awareness of the Department’s shift in policy emphasis. 


Industry’s awareness of existing schemes 


The National Audit Office survey in 1994 asked companies whether 
they had heard of the Department’s existing schemes for stimulating 
industrial innovation, prompting them with the names of the 
schemes. It was likely, however, that fewer would have heard of RIN 
than the other technology development schemes because it is only 
available within designated geographical areas. Although 
Biotechnology Means Business should have been known to 
biotechnology companies there is no reason why general 
manufacturing companies should be aware of it. Also, 
Manufacturing Intelligence was of interest only to a limited part of 
the manufacturing sector. These factors are important in 
considering the results set out below. 


Figure 5 shows that, with the exception of SMART, less than half of 
companies applying to a given scheme were aware of the 
Department’s other schemes. This was irrespective of the size of the 
company or whether they were from the biotechnology or 
manufacturing sectors of industry. These findings were echoed in a 
survey undertaken by the British Chambers of Commerce in 1994. 
This found that well over half of their sample of 394 small firms 
were unaware of the Department’s schemes open to them. 
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Figure 5: Awareness of schemes amongst developers and users 


Non-applicants to 

any of the four 
technology development 
schemes examined by 
the National Audit Office 


scheme 


Users Awareness figures 
for each scheme 
exclude those who 

| Applicants 3 participated in it. 


40 


percentage aware 


Source: National Audit Office survey. 


Figure 5 shows the awareness of the Department's schemes amongst the 510 developers and 150 
users interviewed as part of the National Audit Office survey. 


2.9 As is to be expected, companies who were developers of 
technologies were more aware of technology development schemes 
than of technology transfer or best practice awareness schemes. 
More surprisingly, the same was true generally of users of 
technology who were no more aware of these schemes than 
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developers. They might be expected to have a greater awareness of 
the technology transfer or best practice awareness schemes which 
are designed to be more directed towards their needs. 


The level of awareness amongst non-applicants was low generally. 
In the case of SPUR, the level of awareness was only 9 per cent. 


Of the 402 technology developers surveyed who had made 
applications, over a quarter had been involved in more than one 
initiative over the three years to December 1993, possibly indicating 
a level of recycling of existing applicants. 


Although there was a relatively high level of awareness amongst 
industry of the SMART scheme, industry’s level of awareness was 
low for the other seven innovation schemes as at mid 1994. This low 
awareness, and the possible recycling of existing applicants for 
technology development schemes, introduces the risk that the 
Department are not always securing the best or most relevant 
applicants or participants. In the view of the National Audit Office 
the Department could significantly reduce this risk by spreading the 
knowledge of their schemes more widely. The costs of doing so will 
represent a small proportion of a scheme’s overall costs and are 
likely to be outweighed by the benefits derived from attracting a 
greater number of the best and more relevant applicants or 
participants. 


The Department expect the publication of the 1994 Competitiveness 
White paper, their increased use of industrial secondees and the 
increasing network of Business Links, to have increased industry’s 
awareness of their innovation schemes since the National Audit 
Office survey in 1994. The National Audit Office recommend that the 
Department should monitor closely the impact of these measures in 
securing increased industry awareness of their innovation schemes. 


Industry’s awareness of the difference between 
technology development; technology transfer and best 
practice awareness 


The difference between the concepts of technology development, 
technology transfer and best practice awareness is not easy to 
grasp. It is important, however, that the distinction is understood by 
potential applicants if they are to benefit from participation in 
schemes which are most likely to meet their needs. The National 
Audit Office therefore examined whether industry is aware of the 
difference between the concepts. 


on 


Figure 7: Sources of awareness 
of policy change 


sources of awareness 


percentage of those aware 


Source: National Audit Office survey. 


Figure 7 shows the sources of awareness 
of the policy change amongst the 
companies aware interviewed as part of the 
National Audit Office survey. 
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Of those surveyed in 1994, 30 per cent claimed to be fully aware of 
the differences with a further 26 per cent being partially aware. 
Over two-thirds of the respondents to the survey were unaware of 
the difference between technology transfer and best practice 
awareness. There was no Statistically significant difference between 
developers and users of technology, or between those who had 
applied to the Department and those who had not. This latter 
finding is of concern since it demonstrates that the Department have 
not succeeded in clarifying the distinction for companies who are in 
contact with them. The Department have subsequently 
re-designated both technology transfer and best practice awareness 
as technology access (see paragraph 1.8). 


Industry’s awareness of the shift in policy emphasis 


The National Audit Office found that 20 per cent of those surveyed in 
1994 were aware of the shift in policy emphasis and of these just 
over half claimed to be aware of why the Department had made the 
change. Figure 6 shows how awareness varied between the different 
groups surveyed and Figure 7 the sources of their information. 


Figure 6: Awareness of policy change amongst developers 
and users 


Aware of policy change 


ie Aware of why change made 


percentage aware 


Non Users Overall 
Applicants 


scheme 
Source: National Audit Office survey. 
Figure 6 shows the awareness of the Department's change in policy emphasis amongst the 660 


companies interviewed as part of the National Audit Office survey. In particular it shows the low 
awareness overall. 


22 


The Department of Trade and Industry’s 
Support for Innovation 


2.17 Recent measures, such as the introduction of Business Links, should 
help to raise industry awareness of the Department’s policies and 
schemes. But the Department should monitor industry awareness 
and track the success of recent initiatives in securing higher 
awareness. 
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Part 3 : The framework 
for and outcome of the 
Department’s Support 
for Innovation 


The National Audit Office examined the key elements of the 
framework set by the Department for their support for innovation 
and the outcome of that support. 


The Department’s Innovation portfolio 


The inter-relationships between the schemes in that part of the 
Department’s innovation portfolio examined by the National Audit 
Office is illustrated in Figure 8. This shows the cross sector overlap 
between SMART and RIN which results in failed SMART applicants 
in certain designated geographical areas being automatically 
considered for RIN. It also shows the sector specific overlap between 
the Managing in the 90s and Biotechnology Means Business schemes 
which reflects the support both schemes provide for raising 
industry’s awareness of best practice. 


At the time of the National Audit Office fieldwork in early 1994, the 
Department had in place arrangements to review the overall 
balance of their portfolio of innovation schemes through their 
annual forward planning round. Subsequently, the Department have 
introduced measures to improve their scheme portfolio design and 
resource allocation. 


The revised system first involves identifying the overall gaps and 
overlaps in their portfolio of schemes by taking account of their 
character as either “horizontal” or “vertical” in nature in the 
manner illustrated in Figure 8. The position of both new scheme 
proposals and existing schemes within this framework will be 
identified and each will be systematically assessed against policy 
criteria and awarded merit values which will help to determine 
resource allocations. The Department are also considering adoption 
of the methodology used in the National Audit Office work on 
comparative cost effectiveness to assist with resource allocation. 
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Figure 8: Interrelationship between the selected innovation schemes examined by the 
National Audit Office 
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Specific Technology 
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Source: National Audit Office. 
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(sector specific) 
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Figure 8 shows the sectoral interrelationship between the eight selected innovation schemes examined by the National Audit Office. In 
particular it shows the overlap between SMART and RIN in those designated geographical areas for which funding is available under RIN. 


This overlap is however non-sector specific. 


ae) 


Rationale for the schemes 


As part of the arrangements for design of a scheme, the Department 
examines the economic rationale for the scheme. Pre-conditions for 
intervention include the identification of a significant market failing 
within the Department’s remit for action; the possibility of a scheme 
being devised to stimulate additional activity which will combat the 
failing; and the ability of participants to exploit the outcomes of 
intervention. This process includes an assessment of the 
characteristics of the companies to be targeted in terms of their size 
and industrial sector. The rationale is set out in a document known 
as a ROAME statement, which also specifies the objectives and 
targets for the scheme and the arrangements for project appraisal, 
on-going monitoring and the evaluation of the scheme. 
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3: 6 The National Audit Office noted that, in general, there was an 
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adequate assessment of the economic rationale for all eight schemes 
they examined. Table 2 below and Figure 8 show the key elements 
of the rationale for these schemes in terms of the characteristics of 
the target companies and technology sector. 


Table 2: Size and type of firms targeted 


Target Firms 


Scheme Company Size Company Type 
LINK Non specific Developer 
SMART Small Developer 
SPUR Small & medium Developer 
RIN Small Developer 
Biotechnology Small & medium User 
Means Business 
Manufacturing Medium 
Intelligence ay 
Managing in the Majority small Non specific 
90s (phase 2) or medium 
Teaching Majority small Non specific 
Company or medium 


Scheme 


Source: The Department's ROAME Statement for each of the Schemes. 


Table 2 shows the characteristics of the firms to be targeted by each scheme. 


The precise design and specification of schemes must be based on 
forecasts of future industry status and needs, drawn from current 
information that is not of itself comprehensive or ideal for the 
purpose. Monitoring and evaluation of schemes as they progress are 
therefore necessary ingredients to help counter initial uncertainties 
of design, as well as to reflect changed external circumstances. Two 
of the schemes examined illustrate the difficulties of maintaining 
good scheme design. 


For both Managing in the 90s and the Teaching Company Scheme, 
monitoring and evaluation activity has revealed the need to target 
particular audiences within overall scheme boundaries to secure 
best value for money. It is notable that only the more recent 
monitoring and evaluation activities have revealed the need for 
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change. This reinforces the desirability of the improvements the 
Department are making to their monitoring and evaluation 
processes. 


Objectives and whether they have been met 


A key feature of scheme design is the extent to which objectives are 
set which are both commensurate with and provide adequate 
coverage of the schemes’ aims. Performance against specific 
objectives and targets should also be readily measurable and, where 
it is practicable to do so, objectives should be output orientated. The 
National Audit Office analysed these aspects of scheme design for 
the eight schemes covered by this report. They also assessed 
whether performance against objectives had been measured and 
whether the stated specific objectives had been met. Objectives and 
targets for each of the eight schemes are set out in Appendix 3. 


Compatibility with Departmental objectives and 
coverage of scheme aims 


Save for the LINK scheme, the National Audit Office found that 
objectives and targets were set which were commensurate with 
Departmental objectives. They also judged that the objectives 
provided average to good coverage of scheme aims. 


For LINK, although general overall scheme objectives were set and 
each programme within the scheme has its own objectives, the 
Department had not put in place more specific overall objectives for 
the portfolio of LINK programmes they support. 


Output orientation and measurability of objectives 


In overall terms the National Audit Office judged the objectives and 
targets to be of average output orientation and good measurability. 
There were, however, significant variations between technology 
development on the one hand and technology transfer and best 
practice awareness on the other as set out below. 


Although some specific output orientated objectives and targets had 
been identified in all cases except for the Teaching Company 
Scheme, the balance of coverage for technology transfer and best 
practice awareness schemes was predominantly input orientated. 


The relative lack of output orientated objectives for technology 


transfer and best practice awareness can, to some extent, be 
attributed to the Department’s difficulty in defining readily 


val 


Figure 9: Projects completed 
by scheme 
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Source: National Audit Office survey. 


Figure 9 shows the percentage of those 
companies in receipt of funding that had 
completed their projects and the percentage 
whose projects had been extended or failed. 
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measurable output measures for such schemes. Such outputs are 
usually “soft” in nature and more difficult to isolate and attribute to 
the Department’s support, whilst outputs for technology 
development are usually identified more readily in terms of projects 
completed and commercially exploited. 


The measurability of objectives for the four technology transfer and 
best practice awareness schemes were judged by the National Audit 
Office to be good, compared with average for the four technology 
development schemes. This is attributable to the predominance of 
input objectives under the technology transfer schemes as these are 
typically easier to measure. 


Achievement of objectives 


Most measurable objectives had been measured and virtually all had 
been achieved. In cases where such objectives have not been 
measured this is generally because it is too early to do so. 


Performance indicators 


Although the Department maintain a wide range of performance 
indicators on a scheme by scheme basis these are predominantly 
input based, covering aspects such as the number of grant 
applications received or the number of participants in technology 
and best practice events. Output data are collected as part of the 
formal evaluation system but not for all schemes nor all aspects of 
an individual scheme. In the view of the National Audit Office output 
data need to be collected more systematically and performance 
indicators need to be improved, in particular more output orientated 
ones should be developed. They note that the Department’s revised 
evaluation arrangements (paragraph 4.3) together with the work on 
comparative cost effectiveness should provide a mechanism for 
improving the quality of output data and performance indicators 
generally. 


Outcome of support for schemes: 
technology development 


Projects completed 


The National Audit Office examined the completion rate for projects 
within each of the four technology development schemes. In their 
survey they asked developers who had received funding from the 
Department whether their project had been successfully completed. 
Figure 9 shows the percentage of projects completed on schedule 
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Figure 10: Commercial exploitation 
by scheme 
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Figure 10 shows the extent to which 
companies funded by the Department and 
interviewed as part of the National Audit 
Office survey had commercially exploited 
or patented their projects. In particular it 
shows the high commercial success rate 
for SPUR and the high patenting rate for 
LINK and SMART. 
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and those which had failed or been extended under each scheme. 
This shows that RIN had the highest rate of completions on schedule 
(77 per cent) whilst SMART had the lowest (54 per cent). It should 
be borne in mind that project extensions do not generate additional 
costs for the Department. 


Commercial exploitation and patents 


Two measures of a successful outcome of a completed project are: 
its commercial exploitation; and the registration of a patent, 
although either might take many years to achieve from the inception 
of the project. 


The National Audit Office asked developers who participated in their 
survey whether they perceived their project to be near market, with 
an identified commercial prospect, or far market. Over 80 per cent 
of respondents saw SMART, SPUR and RIN projects as near market 
whereas only just over half saw LINK projects as near market. This 
latter result is consistent with the pre-competitive nature of the 
LINK scheme. However, the average expected time to commercial 
potential being realised from the start of the project was 
surprisingly low: nearly four years for LINK; nearly three years for 
SMART and SPUR; and two years for RIN. 


Of projects completed, over half had been commercially exploited 
and a third patented. Figure 10 analyses these results by scheme. It 
shows that SPUR has the highest rate of commercial exploitation whilst 
LINK has the lowest. SMART has the highest rate of patenting whilst 
SPUR has the lowest. These results are compatible with the stated 
objectives of the schemes in terms of the distance from market at 
which the schemes operate. The National Audit Office noted that the 
survey results for commercial exploitation are broadly in line with the 
Department’s own evaluation results for these schemes. 


Additionality and deadweight 


The National Audit Office examined the following two key measures 
of the outcome of the Departments support: 


e additionality - an activity is additional if it would not have taken 
place in the absence of Departmental support. There are two 
levels of additionality. The first of these is the extent to which the 
scheme as a whole is additional and the second is the extent to 
which individual projects are additional; and 
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Figure 11: Additionality by scheme 
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Figure 11 shows the percentage of 
companies in receipt of funding from the 
Department that perceived that they would 
not have gone ahead without funding or 
“additionality". In particular it shows the 
high additionality for LINK and the 
relatively low additionality for SPUR. 
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e deadweight - the difference between the actual funding received 
and the minimum necessary to encourage companies to proceed 
with the project. 


Additionality 

The overall additionality of a scheme is a key aspect of its rationale - 
that is whether the scheme is providing a necessary market need 
which the market itself is unable to provide (see paragraph 3.5). The 
Department therefore attach equal importance to overall scheme 
additionality in all schemes. However, the extent to which the 
Department use the application appraisal process to test the 
expected additionality for individual projects is determined by the 
balance between the size of the grant and the administration costs 
associated with application appraisal activity. Accordingly, expected 
additionality is a key criterion in assessing applications for the LINK 
scheme, which aims to bring together a consortium from academia 
and industry to develop innovative technologies, and also for 
applications under the SPUR scheme. The Department attach less 
importance to additionality in assessing applications for SMART and 
it was not a criterion in their selection of RIN projects. 


Three-quarters of developers in receipt of Departmental funding 
under the four technology development schemes and surveyed by 
the National Audit Office perceived that they would not have 
proceeded with their project without funding from the Department. 
Figure 11 shows the range of responses by scheme. It should be 
noted that, under the definitions adopted, a project could rate as 
additional even if the company received more funding than was 
necessary to stimulate the project. 


The lower additionality for SPUR than for SMART or RIN appears to 
be at variance with a key criterion for that scheme. However the 
SMART and RIN schemes are targeted at smaller companies than is 
SPUR; and grant paid under those schemes is likely to represent a 
higher proportion of company turnover (Figure 12). That in turn 
makes it more likely that companies could not proceed without 
Government support, because of difficulties in raising the necessary 
finance. Moreover, an evaluation report by the Department showed 
that a significant proportion (28 per cent) of SPUR proposals would 
have been delayed or extended without Departmental support. This 
“partial additionality” is greater in the case of SPUR than other 
schemes. The National Audit Office noted that these findings are 
broadly comparable with the Department’s evaluation of technology 
development schemes generally. 
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Figure 12: Percentage of turnover represented by grant 
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Source: National Audit Office survey. 


Figure 12 shows the percentage of turnover represented by grant for the 
289 companies in receipt of funding interviewed as part of the National Audit Office survey. 


Table 3: Overall Deadweight Deadweight 
by Scheme” : ; j Cas 
Total Sucvayon TEetneea en 3.26 As part of their survey the National Audit Office asked companies in 


Scheme Deadweight eee aa receipt of a grant what was the minimum level of funding they 
would have required to have gone ahead with the project. The 
stated minimum was then compared with the funding received to 
see if the Department were providing grants in excess of those 
SPUR necessary to ensure the projects proceeded, that is “deadweight”. 
RIN Table 3 shows the total deadweight identified in the survey and the 
National Audit Office’s estimate of total annual deadweight and 
Figure 13 overleaf shows the average amount of deadweight by 
scheme. There is a general risk in surveys of this type that 


MINS 


SMART 


Notes: (1) See note (1) to Figure 13. 
(2) Estimated overall annual 


deadweight is based on average respondents are likely to be overly conservative in their estimate of 

programme expenditure over the mini . c Hen dweicht is likelv to be 

three years to March 1993. LINK the eee funding required. ence dea 8 y 

should be treated with caution for greater than that reported. 

the reasons set out in paragraph 

3.27. me é ; ; Ae 

3.27 A number of participants in the survey, particularly universities and 

Source: National Audit Office. large manufacturing companies who had received grants under the 
Table 3 shows the significant variation in the LINK programme, said they would have needed see grants to 
overall deadweight between the schemes. proceed although, in fact, they had gone ahead with their projects. 


Their responses have been excluded from the analyses in Table 3 
and Figure 13. Furthermore, some of the companies or universities 
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Figure 13: Deadweight by Scheme(1) 
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Source: National Audit Office survey. 
Figure 13 shows the significant variation in 
average deadweight between the schemes. 


In particular it shows the apparently high 
deadweight for LINK compared with RIN. 
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surveyed who had received grants under the LINK scheme may have 
been referring to the total project cost rather than their own share 
of it: the results for LINK should therefore be treated with caution. 


Because of the definition adopted for additionality and deadweight a 
project which was additional can also carry deadweight if the 
company received funding in excess of the minimum required to 
proceed. It is therefore possible for a scheme to have both high 
additionality and high deadweight, as is the case for LINK. 


Outcome of support for technology transfer 
and best practice awareness 


Impact of participation in schemes 


The National Audit Office surveyed companies who had participated 
in technology transfer and best practice schemes and asked whether 
changes had been made to their business as a result of their 
participation in Departmental schemes, and if any such changes had 
made them more competitive. Their responses are analysed in 
Figure 14 opposite. This shows that some 60 per cent of participants 
found the events suitable for their needs and that 30 per cent of 
participants had taken action as a result. In the majority of cases the 
companies concerned perceived that the changes had made them 
more competitive. Typical responses are shown in the box. 


“We gained knowledge in the selection of equipment. We now know 
better what we are looking for.” 


“We got interesting ideas on how to exploit our technologies.” 


“(It) has improved management skills in both communication and 
awareness of commercial issues.” 


The 1994 survey findings in Figure 14 represent worthwhile results 
which are broadly in line with typical Departmental objectives for 
such schemes. They are also in line with the results of a 
Departmental evaluation in 1992 - which suggests the Department 
will need to take specific action if they wish to raise levels of 
relevance and successful outcomes. 


The National Audit Office identified two sorts of improvements 
which could be made: 
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Figure 14: Results of participation 
in technology transfer 
and best practice 
schemes 
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Source: National Audit Office survey. 


Figure 14 shows the suitability ratings 
achieved by technology transfer and best 
practice schemes and the worthwhile 
results for improved competitiveness 
ratings. 
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e greater awareness of the schemes could enhance their cost 
effectiveness by widening appropriate participation and thus 
spreading costs over a greater number of participants; and 


e for the two technology specific schemes, Biotechnology Means 
Business and Manufacturing Intelligence, a significant minority of 
participants were not drawn from the prime target audiences. 
The cost effectiveness of these schemes could have been 
improved if the selection process for participants were better 
defined (paragraph 4.16). 


Admission costs to events 


Participants in technology transfer or best practice schemes 
examined by the National Audit Office were not required to pay for 
admission save for a fee of up to £60 for participating in certain 
Managing in the 90s seminars. In their survey the National Audit 
Office asked companies whether they would be willing to pay to 
participate in such events. Around half said they were prepared to 
pay to participate in technology transfer events and two-thirds to 
participate in best practice initiatives. The average amount 
companies were willing to contribute was about £170 per event. 
Following the results of the National Audit Office survey the 
Department are considering appropriate admission fees for these 
events. The National Audit Office welcome this. 
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Part 4: Evaluation of 
schemes and their 
comparative cost 
effectiveness 


The National Audit Office examined the Department’s evaluation 
process to determine whether it provided an adequate mechanism 
for judging the Department’s performance. They also considered the 
relative cost effectiveness of the Department’s schemes. 


Evaluation 


The Department aim to undertake formal evaluations of all major 
expenditure schemes to inform policy making and programme 
management on a scheme by scheme basis. Most of the schemes 
considered in this report have been recently evaluated in full. The 
exceptions to this are: LINK, for which a number of individual 
programmes within the scheme have been evaluated; and RIN which 
was last evaluated in 1991. 


Recognising the need to adopt existing best practice more widely, 
the Department introduced revised evaluation arrangements in late 
1993 which were designed primarily to integrate the evaluation 
process and resource allocation management (see paragraph 3.3) 
with the arrangements for the on-going monitoring of schemes. 
These arrangements should enable the Department to identify and 
collect a standard core of monitoring data and ensure that 
evaluation results are available whilst schemes are in progress, so 
as to achieve the maximum impact on policy and scheme 
development. They should also help to ensure that the rationale and 
objectives for schemes are defined as rigorously as possible in 
advance. Standard detailed evaluations will continue to be carried 
out for some schemes. These revised arrangements are similar to 
those used by the European Commission and will be fully 
operational by 1996. 
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The revised arrangements are primarily concerned with the 
evaluation of individual schemes, although there are provisions 
within them to examine, on an ad-hoc basis, specific issues which 
cut across a number of schemes. Whilst the National Audit Office 
support these arrangements they note that there is no provision in 
them systematically to evaluate the relative cost effectiveness of 
schemes in overall terms. In their view this limits the Department’s 
ability to learn lessons from different aspects of schemes’ design and 
operation and to determine the most cost effective allocation of 
resources between them. 


Comparative cost effectiveness 


The Department formed a joint working group (the Group) with the 
National Audit Office early in 1994 to identify a more comprehensive 
range of indicators than then employed and a methodology for 
assessing the comparative cost effectiveness of the eight selected 
schemes. Detailed terms of reference for the Group are set out in 
Appendix 2. 


The approach adopted by the Group, developed by the consultants 
commissioned by the National Audit Office, is based on an extension 
of decision theory. It is set out in Appendix 2. 


The output from this Group’s work on comparative cost effectiveness 
is not definitive and the results should be interpreted and used with 
caution. It does, however, provide an indication of the relationship 
between the cost of a scheme and the amount of innovation and 
competitiveness it delivers. The methodology therefore provides a 
management tool which could help in diagnosing potential problems 
and pointing to possible courses of action to alleviate them. 


Development of appropriate performance indicators 
Two over-arching objectives were identified: 

e minimising costs; and 

e maximising innovation and competitiveness. 


It is the balance of these that determines the comparative cost 
effectiveness of any particular scheme. To develop performance 
indicators, these objectives were each broken down into a number 
of specific criteria. Each criterion could be measured by 
performance indicators. Seven criteria and 19 performance 


on 


The Department of Trade and Industry’s 
Support for Innovation 


indicators relating to them were developed for the model. Some 
were already used by the Department. The criteria and performance 
indicators are set out in Figure 15 opposite. 


Ranking of schemes 


Figure 16: Comparative scores of 4.9 The Group determined overall scores for the cost effectiveness of 
eight schemes each scheme. The results are set out in Figure 16. This shows that, 
examined by the ‘ i 
National Audit Office overall, the Teaching Company Scheme is most preferred (61) 
ESSE: followed by RIN (57). LINK is the least preferred (25). 


4.10 To gain a better picture of cost effectiveness, the scores for “costs” 
and “innovation” need to be considered separately; this is shown in 
Figure 17. Schemes lying on the curve can be considered to be 
delivering innovation at relatively low cost; those inside the curve 
may be performing less cost effectively. Figure 17 shows that all of 
the schemes save LINK lie on or just inside the curve. 
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Source: The Group. 


Figure 16 shows the scores achieved by 
the eight schemes examined by the 
National Audit Office under the comparative 
cost effectiveness model. In particular it 
shows the relatively low score achieved by 
LINK compared with the other seven 
schemes examined. 
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4.11 By carrying out a sensitivity analysis on these results and drawing 
upon the relevant findings set out in the earlier parts of this report 
the National Audit Office examined further: 


e the effect of changes in the balance between cost and innovation 
on the ranking of schemes; 
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Figure 15: Hierarchical representation of performance criteria and indicators 


Performance Performance Definition of 
Costs and Benefits Criteria Indicators Performance Indicators 


number of licences issued 


commitment to exploit the product 


number of products to market 
within 36 months 


: OM'NT Ri commitment to Research and 
R&D : Development 


number of completed projects 


number of organisations continuing 
to collaborate 


number of significant technological 


INNOVATION enhancements 


AND ACQUISITION 


COMPETITIVENESS number of significant improvements 


in business practice 
number of licences acquired 


number of organisations involved in 
collaborative agreements 


== CONTACTS number of contacts with information 
ACCESS | noms service 


number of organisations party 
OVERALL ae to accepted offers 


number of pre-filtered applications 
AWARENESS | 


number of participants per year 


number of failed applications 


NON-MONETARY average application processing time 


number of outside bodies involved 


Ge PRO! programme costs 
MONETARY —e 
= INNING running costs 


COSTS 


*Identifies the 12 most important indicators (see paragraph 4.14). 


Source: The Group. 


Figure 15 shows the hierarchical structure of the comparative cost effectiveness model. In particular it shows the interrelationship between the individual 
performance criteria and their related performance indicators and the latter's definitions. 
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e the principal factors which have determined the ranking of 
schemes; and 


e how improvements in the schemes might be secured. 


Effect of changes in the balance between cost and 
innovation 


The results shown in Figures 16 and 17 are based upon equal 
weights being assigned to the overall criteria of cost and innovation. 


The effect on these results of changes to this balance between cost 
and innovation are shown in Table 4. This demonstrates that the 
rankings are dependent upon the relative importance placed on 
costs or innovation. As costs become more important technology 
transfer and best practice awareness initiatives are rated more 
highly than grant schemes, with the exception of RIN. However the 
ranking for LINK is unaffected by wide variations in the balance of 
preference for costs or innovation. 


Table 4: Impact of changes in the balance of costs and 
innovation 


Programme Overall weight on costs 


50% 
Teaching 
Company 
Scheme 
RIN 


Biotechnology 
Means Business 


Manufacturing 
Intelligence 


Managing 
in the 90s 


SPUR 


SMART 


LINK 


Source: The Group and Facilitations Ltd. 

Note: The figures in the table represent the scores for each scheme under two further options: 25% 
weight on costs (low preference) and 75% weight on costs (high preference). Rankings are shown in 
brackets for each scheme. 


Table 4 shows the effect of changes in the balance of preference for costs and innovation on the 
weighted average score for each scheme. 
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Principal factors contributing to the ranking of the 
schemes 


4.14 Sensitivity analysis suggests that 12 of the 19 performance 
indicators will provide a model 96 per cent similar to the one 
developed using all 19. Table 5 illustrates the performance of each 
of the schemes against the 12 key indicators. The Teaching 
Company Scheme achieves its high ranking principally because it 
performs well across a wide range of performance indicators: in 
particular, its unit costs are relatively low; processing time is 


Table 5: Principal factors contributing to the ranking of the schemes 


SCHEME (ranked in order of overall cost effectiveness score) 


Biotech- Manu- 
Teaching nology facturing 
Company Means _ !ntelli- Managing 
Scheme RIN Business gence jn the 90sSPUR SMART LINK 


EREERESeR 
i 
| 
| 


Number of Products to 
market within 36 months 

Number of organisations involved 
in collaborative agreements 


— 


- 


Number of 
completed projects 


Number of significant improve- 
ments in business practice 


Number of significant 
technological enhancements 
Number of organisations 
party to accepted offer 


Number of 
licences acquired 
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Source: Facilitations Limited and the National Audit Office. 


Table 5 shows the “Preference Scores” (see Appendix 2) achieved by each scheme on each of the 
12 most important performance indicators. In particular it shows the relatively poor scores achieved 
by LINK compared with the relatively high scores achieved by the Teaching Company Scheme. 
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relatively short and it finances collaborative agreements. LINK is 
ranked the lowest because although it also finances collaborative 
agreements it does not generally perform well against other criteria: 
in particular the processing time is protracted (see Figure 19), the 
administration costs are high and the rate of commercial 
exploitation is relatively low (see Figure 10). 


How improvements in the schemes might be secured 


The National Audit Office considered by reference to Table 5 how 
improvements in the cost effectiveness of schemes might be secured. 
The scores achieved by schemes under certain performance 
indicators in part reflect the design of the schemes. For instance, the 
relatively low preference score achieved by Managing in the 90s on 
the project and product related performance indicators can be 
attributed to the absence of funded projects under this scheme. 
Similarly, the relatively low commercial exploitation preference 
score achieved by the LINK scheme can, in part, be attributed to the 
distance from market at which this scheme operates. 


Nevertheless, the National Audit Office identified the following key 
points for improving cost effectiveness which have broad relevance: 


e the high weightings attached to programme and running costs 
emphasise the importance of minimising deadweight for 
technology development schemes, and maximising administrative 
efficiency for all schemes. Part 5 explores these points in more 
detail; 


e the relative performance of the LINK scheme is less good in these 
regards: recent Office of Science and Technology proposals (see 
paragraph 5.8) may help with administrative problems, but the 
high deadweight should also be addressed; 


e within the group of innovation indicators, those relating to 
projects and products are weighted higher than those dealing 
with transfer of information; 


e access schemes are likely to do better if they are targeted in 
terms of topic and audiences in a way that maximises the chance 
of changes to business practices or use of technologies. The way 
the Teaching Company Scheme combines development and 
access elements seems well suited to this purpose. Better 
targeting of potential participants is discussed in Part 3; 
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e there appears to be scope to increase ratings for project 
completion and commercial exploitation for SPUR and LINK, to 
bring them close to the levels obtained by the other two 
technology development schemes. Part 5 explores one way of 
achieving this, by improving the quality of application appraisals. 


4.17 In order to get greatest value from their experiences under 
individual schemes, the Department should investigate further the 
use and development of the comparative cost effectiveness 
methodology used in this study. This work would build on the 
Department’s existing evaluation arrangements, and the results 
would feed into budgeting and resource allocation systems. 
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Figure 18: Offer Rates by 
Scheme1 


offer rates (percentage) 


LINK SMART SPUR RIN 


scheme 


Source: National Audit Office's analysis of 
the Department's management 
information. 


Note (1) The offer rate is the percentage 
of grant applications which result in an 
offer of funding from the Department 


Figure 18 shows the differing offer rates 
between the schemes. In particular it 
shows the low offer rate for SMART 
which reflects the competitive nature of 
the scheme. 
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Part 5: Administration 
of the schemes 


The National Audit Office examined the Department's efficiency in 
administering the eight selected schemes. In general they found 
that, at the time of their fieldwork in early 1994, the Department 
administered the schemes well. In particular, grant applications 
were usually appraised in line with laid down procedures and 
scheme eligibility requirements; and the progress of funded projects, 
activities, events and schemes were, in most cases, well monitored. 
However, the National Audit Office noted aspects of the process 
which affected the cost effectiveness of certain schemes, as set out 
in the earlier parts of this report. These are considered in more 
detail below. 


Project appraisal procedures 


Appraisal procedures for applications to the eight schemes 
examined are set out in Appendix 4. 


Appraisal of technology development projects 


Although appraisal criteria are used to judge the quality of 
applications for funding under the SPUR, SMART and RIN schemes, 
applications for the SPUR and RIN schemes are usually funded 
provided they meet the eligibility criteria drawn up by the 
Department. However, for the competitive scheme, SMART, a formal 
marking system is used to appraise the quality of projects and this 
helps to determine which applications will be funded and which will 
be rejected. Figure 18 shows the differing offer rates which result 
from these approaches in the four years to 31 March 1994 and, for 
completeness, the offer rate for LINK. 


In the view of the National Audit Office, a more formalised marking 
system for LINK and SPUR should be introduced because: 


e it could assist in setting a minimum quality standard for 
proposed projects and provide the Department with information 
both to track the quality of industry’s projects and monitor the 
consistency of project appraisal; 
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e better quality projects should also have a greater chance of 
success. This in turn should increase the cost effectiveness of the 
schemes through increases in the rates of project completion and 
commercial exploitation (see paragraph 4.16); and 


e greater awareness of the schemes (paragraph 2.12) would 


generate more applications. The Department would need a more 
robust means of identifying the projects it should fund. 


The processing of applications 


5.5 The National Audit Office analysed the processing time by scheme 


for all applications received in the four years to 31 March 1994. The 
results are set out in Figure 19. 


Figure 19: Grant processing times 


scheme 


processing time (days) 


Source: National Audit Office. 


Figure 19 shows the average time to process a grant application for each scheme. In particular it 
shows the relatively high processing time for LINK compared with other technology development 
schemes. 


5.6 Two-thirds of technology developers receiving funds from the 


DEA 


Department who participated in the National Audit Office survey 
claimed, when prompted, to have experienced problems during 
their negotiations with the Department. A major difficulty they 
experienced was the processing time for applications. This is 
illustrated at Figure 20 overleaf. 


The application process for LINK and RIN were perceived as 
significantly worse than for SMART or SPUR. In the case of RIN, 
which had the shortest absolute time for application processing of 
schemes examined, dissatisfaction is most plausibly explained by the 
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Figure 20: Problems experienced by companies applying for grants 
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Source: National Audit Office survey. 


Figure 20 shows the problems experienced during the application process by the companies applying 
for grants interviewed as part of the National Audit Office survey. 


timescale relative to normal business practice: RIN is targeted at 
small companies undertaking relatively straightforward projects 
where there may be expectations of quick decisions. 


The processing time for LINK project proposals was the longest of 
the technology development schemes examined. This can partly be 
explained by the lengthy internal approval arrangements set out in 
Appendix 4. This finding is supported by a recent review of the LINK 
scheme by the Office of Science and Technology, who confirmed that 
administrative delays are a significant problem for the scheme. They 
have made recommendations designed to streamline the approval 
arrangements set out in Appendix 4 for which the Government is 
committed to their early implementation. These changes should 
reduce both the processing time for the LINK project proposals and 
the administration costs borne by the Department. These in turn 
should improve the cost effectiveness of the LINK scheme (see 
paragraph 4.16). 
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Figure 21: The Department's 
average downward 
adjustment of eligible 
costs 


average downward adjustment (%) 
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Source: National Audit Office. 


Figure 21 shows the Department's 
downward adjustment of companies’ 
assessments of eligible costs. 
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Grant levels for technology development 
projects 


The level of grant received by technology developers is determined 
solely on the basis of eligible costs. For each scheme the Department 
have drawn up a definition of eligible costs. Figure 21 shows the 
impact of the Department’s adjustments to companies’ assessment 
of eligible project costs. Although the average adjustment for all the 
schemes was downwards, with the exception of RIN, the companies’ 
assessments of eligible cost are generally similar to those of the 
Department; the average downward adjustment being well under 

10 per cent. 


For each scheme the Department have designated the percentage of 
eligible costs which they are prepared to fund, which are similar to 
those funded by the European Commission and other countries. For 
all the technology development schemes except LINK the 
Department have also determined a financial ceiling up to which 
they will fund eligible costs. These are set out in Figure 22. 


With the exception of those LINK projects whose eligible costs 
exceed £0.5 million, no discretion is given in the schemes examined 
by the National Audit Office to vary the percentage of eligible costs 
funded. The guidelines leave little room for flexibility in assessing 
the most effective level of funding within a ceiling and thereby the 
level of grant which should be awarded. This may contribute to the 
extent of deadweight noted in paragraphs 3.26 and 3.27, which 
impacts on the cost effectiveness of the LINK, SPUR and SMART 
schemes. The deadweight results for individual schemes suggest 
that the mechanism of fixed grant levels adopted by the Department 
contributes to lower deadweight for schemes targeted at smaller 
companies and small and straightforward projects. 


5.12 Although it is not possible to eliminate deadweight by scheme 


ao 


design, the Department might consider adopting more flexible 
funding within a ceiling for medium sized and larger companies and 
those projects involving more uncertainty to facilitate a reduction in 
deadweight and hence an improvement in the cost effectiveness of 
the schemes (see paragraph 4.16). 


Costs of administration 


The actual administrative costs associated with the eight schemes 
examined are not readily available since the Department’s costing 
system, MINIS, does not allocate costs to individual schemes. 
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Figure 22: Grant levels by scheme 
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Source: National Audit Office. 


Note: The two sets of bars for SMART represent the details for the two stages of SMART, stage 1 and 
stage 2 (see Figure 3). No separate analysis of the average grant awarded is available. 


Figure 22 shows the maximum grant available under each scheme and the average grant awarded. It 
also shows the maximum percentage of eligible costs the Department is prepared to fund under each 


scheme. 


Estimates of such costs have been made with the help of the 
Department, and are set out in Table 6 opposite. Accurate 
information on the overall costs of the schemes is an important 
element of an effective management information system. 
Accordingly, the Department should consider adapting their existing 
system to generate the necessary cost information to determine 
more accurately the overall costs of each of their schemes. 


5.14 Administrative costs for Managing in the 90s appear high compared 
with programme costs. However, unit costs are low because of the 
relatively high level of participation. In comparison, unit costs for 
LINK are high because administrative costs are high, partly because 
of the nature of the process set out in Appendix 4, and also because 
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the number of grant applications is relatively low. The level of 
administrative costs for LINK contribute to that programme’s 
relatively low cost effectiveness. 


Table 6: Unit costs by scheme 


Scheme Annual Programme Annual Administrative Total Annual Costs Annual Number of Overall Annual Unit Costs* 


Costs' £000 Costs? £000 £000 Applicants or Participants * Admin £000 Total £000 


LINK 15,600 5,000 20,600 315 15.9 65.4 
SMART 12,500 1,000 13,500 3,455 0.3 3.9 
SPUR 7,300 1,000 8,300 240 4.2 34.6 
RIN 5,000 500 5,500 575 0.9 9.6 
Biotechnology 160 40 200 740 0.1 0.3 
Means Business 
Manufacturing 300 44 344 1,015 0.1 0.3 
Intelligence 
Managing in the 3,100 1,000 4,100 15,000 0.1 0.3 
pee yy) 
Teaching 5,000 250° 5,250 1,250 0.2 4.2 


Company Scheme 


Source: The National Audit Office's analysis of the Department's financial and participants records for the three years to 31 March 1994 and discussions with 
the Department. 


Notes: (1) These costs include the costs of consultants employed by the Department to manage certain programme elements and other direct costs of the 
scheme. 


(2) Staff costs, including Regional Office elements, attributable to the scheme as estimated by the National Audit Office and the Department. 


(3) Average number of grant applicants to technology development schemes or participants in Department sponsored technology transfer or best 
practice events or activities. 


(4) Estimated on the basis of costs per technology development grant applicant or technology transfer/best practice participant. 


(5) New liaison arrangements for the Teaching Company Scheme will increase administrative costs borne by the Department. 


Table 6 shows the significant variations in the overall unit costs of the schemes examined by the National Audit Office. 
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Appendix 1 


Survey of industry 


Background 


The National Audit Office commissioned IFF Research Limited to 
undertake a telephone survey (the Survey) of industry to ascertain: 


e industry’s views on and awareness of the change in policy 
emphasis and their awareness of the Department’s schemes; 


e industry’s view of the Department’s management of the schemes; 
and 


e the outcome of industry’s participation in the Department’s 
schemes. 


Key results are included in Parts two to five of this report. 


Methodology 


The Survey was designed to obtain the views of companies from the 
advanced manufacturing and biotechnology sectors falling into the 
following three categories: 


a) suppliers (or developers) of technology who had applied to the 
Department for financial assistance under one of four grant 
schemes during the period April 1990 to December 1993; 


b) developers who had not so applied; and 


c) companies who were customers (or users) of technology 
developed by companies of the type described in categories (a) 
and (b) above. 


In consultation with the Department, telephone questionnaires were 
finalised following exploratory telephone interviews. The telephone 
interviews were carried out by 42 IFF Research executive 
interviewers in the period March to July 1994 among senior 
technical managers in the three categories of companies set out 
above. The survey was stratified both by technology sector 
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(advanced manufacturing and biotechnology) and company size. The 
details of the sample structure and source by sample category are 
set out in Table 7 below. 


Table 7: Sample structure and sources 


Company Size" 


Sample Category 


Developers - 402 314 88 346 56 The Department's 
Applied record of grant 
applicants. 
Developers - 108 71 37 72 36 Dun & Bradstreet 
Not Applied 
Users - Pure 75 56 19 43 32 Dun & Bradstreet 
- Booster (ee 50 25 47 28 The Department's 


record of participants 
_ in technology transfer 
150 106 44 90 60 and best practice 
awareness schemes 


OVERALL 660 491 169 508 152 


Notes: (1) Company size by number of employees 
Source: IFF Research Limited 


Table 7 shows the number of companies responding to the telephone survey according to their sample category, technology sector and size. 


5 The telephone survey was complemented by in-depth interviews 
with 20 companies. These provided evidence to amplify and 
illustrate the statistical findings of the Survey. 


Statistical validity 


6 Survey interviews were carried out on a geographically random 
basis and 1,105 companies were initially contacted for interview 
having been randomly selected from the sources described in 
Table 7 above. The 660 companies interviewed represented a 
response rate of 73 per cent after taking account of those which did 
not qualify for interview or could not be contacted. The likelihood of 
non-response bias is reduced with such a high response rate and 
IFF Research Limited believed the sample to be representative. 
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7 The sampling errors for the overall survey sample are estimated to 
be less than + or - 3.0-4.0 per cent at the 95 per cent confidence 
level. The estimated sampling errors for the three sub-categories at 
the same confidence level are set out in Table 8. 


Table 8: Sampling Error 


Estimated 
Sample Category ad eror 
oO 


Developers - Applicants +or-4 tos 


Developers - Non +or-8 to 10 The survey 
Applicants 
8 Material covered by the Survey included: 
Users +or-6to8 
Se ne e industry’s awareness of, and views on, the changes made to the 


an Department’s support for innovation; 
Source: IFF Research Limited 
Table 8 shows the sampling error both for e industry’s awareness of, and understanding of, the technology 
the main sample categories and the overall 
sample: development and technology transfer schemes, and the 
distinction between them; 


e the appropriateness, efficiency and adequacy of the Department's 
grant application procedures, and how they might be improved; 


e use of European Commission grants; 


e the appropriateness of grant eligible and non-eligible project 
costs; 


e the additionality of the Department’s support and whether 
funding is at the minimum effective level; 


e reasons for Department funded projects not being completed or 
for non-participation in a technology transfer or best practice 
scheme; 


e the extent to which direct funding of projects results in increased 
competitiveness through commercial exploitation of the product; 


e the extent to which participation in technology transfer and best 
practice awareness results in increased competitiveness through 
implementation or adaptation of the technology or best practice; 
and 


e whether the Department’s technology development or technology 


transfer schemes are targeting the technologies that the market 
needs. 
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Appendix 2 


The comparative cost effectiveness of the 
Department’s innovation schemes 


Terms of reference 


Recognising the weaknesses in their existing evaluation system 
(paragraph 4.3) the Department formed a joint working group (the 
Group) with the National Audit Office early in 1994. The Group was 
tasked with developing appropriate performance indicators and a 
methodology for measuring the comparative cost effectiveness of the 
Department’s innovation schemes in delivering their objectives of 
stimulating industrial innovation and increasing the competitiveness 
of United Kingdom industry. The methodology developed was 
applied to the eight schemes described in Figure 3 and involved the 
following tasks: 


a) the identification of criteria which can be used as the basis for a 
system of performance indicators for determining the cost 
effectiveness of the Department’s innovation schemes; 


b) the examination of the extent to which the criteria and 
performance indicators referred to above properly reflect the 
performance of the Department’s innovation schemes in terms of 
cost effectiveness; 


c) the devising of a methodology for assessing the comparative cost 
effectiveness of the Department’s innovation schemes, 
incorporating the identified criteria and performance indicators; 
and 


d) the assessment of the comparative cost effectiveness achieved by 
the Department’s innovation schemes using the methodology 
devised in (c) above. 


The National Audit Office commissioned Facilitations Limited and an 


academic consultant from the London School of Economics to assist 
in this exercise. 
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Methodology 


The approach adopted by the Group was based on an extension of 
decision theory which allows options, such as innovation schemes, 
to be evaluated and compared when they differ on evaluation 
criteria, such as performance indicators. 


Performance criteria and indicators 


Two over-arching objectives were identified: 
e minimising costs; and 
e maximising innovation and competitiveness. 


It is the balance of these that determines the comparative cost 
effectiveness of any particular scheme. To develop performance 
indicators, these objectives were broken down into a number of 
different performance criteria. Each criterion could be measured by 
performance indicators. Seven criteria and 19 performance 
indicators relating to them were developed for the model. The 
criteria and performance indicators are shown in Figure 15, which 
for convenience is repeated (opposite). 


Comparative cost effectiveness 


The evaluation of an individual scheme’s relative cost effectiveness 
comprised four stages. The first used available data to assign values 
to the schemes for each performance indicator. 


In the second stage, the values on a given indicator were 
transformed onto “preference scales”. These are relative scales with 
a preference score of 100 representing the most preferred scheme 
in terms of a particular indicator and a preference score of 0 
representing the least preferred scheme. It should be emphasised 
that these scores are relative values: 0 does not necessarily mean no 
effectiveness, it simply means the lowest performance on a 
particular indicator relative to the other schemes. The remaining 
schemes were usually placed on each preference scale as a linear 
function of their performance indicator value (see Figure 23 on 
page 54). This means, for instance, that a scheme whose 
performance on a particular indicator was half way between the 
least and most preferred scheme was assigned a preference score of 
50 on that indicator’s preference scale. 
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Figure 15: Hierarchical representation of performance criteria and indicators 
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“Identifies the 12 most important indicators (see paragraph 4.14). 


Source: The Group. 


Figure 15 shows the hierarchical structure of the comparative cost effectiveness model. In particular it shows the interrelationship between the individual 
performance criteria and their related performance indicators and the latter's definitions. 
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Figure 23: Conversion of performance value to preference value 


Preference (2) 


Linear Preference 
Non-linear Preference 


Value (1) 


Source: National Audit Office. 


Notes: 

(1) The horizontal axis represents the value of the indicator for each of the schemes. Value X 
represents the scheme with the lowest value on this particular indicator whilst value Y represents 
the scheme with the highest value on this indicator 

(2) See paragraph 6 above. 


Figure 23 shows in general terms the options for converting the value of each scheme on each 
indicator to a preference value . This can be done linearly, as represented by the straight line, or 
non-linearly, as represented by the curved lines. In most cases the Department's schemes were 
placed on the preference scale for each indicator as a linear function of their performance indicator 
value (see paragraph 6 above). 


7 The next stage required each performance indicator’s preference 
scale to be weighted by the Group so that each scale could be 
directly compared with another. This involved assigning a weight of 
100 to the indicator with the most preferred range of values. The 
remaining indicators were then assigned weights relative to this on 
the basis of the range of values for each indicator and the Group’s 
preference for that range compared to the ranges on other 
indicators. This process was carried out separately for the cost and 
benefit side of the hierarchy. 


8 Finally, the weighted preference scores were combined to give an 
overall weighted average score for each scheme, based on equal 
weights (50/50) assigned to the overall objectives of costs and 
innovation. 
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Analysis of comparative cost effectiveness 


The overall comparative cost effectiveness of the schemes was 
assessed by comparing the scores achieved by each. The particular 
aspects of performance under which individual schemes were 
performing well or poorly relative to the other schemes were 
identified by examining the scores achieved by each scheme on 
individual indicators. 


Possible improvements to the schemes were identified by varying 


the values of the indicators and plotting the changes in the overall 
weighted average score. 
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Appendix 3 


Overall and Specific Objectives by Scheme 


Scheme The Scheme’s Overall Objectives The Department’s Specific Primary 
Objectives and Non-Financial Targets 


LINK To help the United Kingdom exploit Each LINK programme (see Appendix 4) has its 
developments in science and technology, own set of objectives. However the Department 
bridging the gap between industry and the had not put in place specific overall objectives 
research community. Specifically LINK aims to: for the portfolio of LINK programmes that they 

support. 


e provide a framework for collaborative 
research programmes and projects in key 
areas of science, technology and 
engineering; 


e — enable and accelerate the commercial 
exploitation of science and technology, 
leading to new products, processes, 
systems and services; 


e promote a close interaction between 
industry and the research base, so that 
nationally supported programmes of basic 
research are influenced by awareness of 
the needs of industry; 


e —_use the research base effectively and to 
increase UK industrial competitiveness; 
and 


e _ stimulate industry to increase its own 
investment in R&D. 


continued 
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Overall and Specific Objectives by Scheme 


The Scheme’s Overall Objectives 


To stimulate small firms to develop and market 
new science and technology based products; to 
encourage and facilitate the formation of viable 


and durable, science and technology based 
small firms; and to contribute to a climate 
which encourages investment in highly 


innovative technology by individuals, firms and 


financial institutions. 


To encourage more small and medium sized 


enterprises (SMEs) to undertake more research 


and development and develop new products 


and processes to benefit the national economy. 


ane 


The Department’s Specific Primary 
Objectives and Non-Financial Targets 


to stimulate at least 25% more worthwhile 
ideas per year meriting a Stage | award 
than there are awards available; 


by the end of each Stage II, to have caused 
at least 10% of the Stage | winners in that 
competition to be viable small firms 
established or registered after winning; 


to stimulate at least one-third of Stage | 
winners to market a SMART-derived 
product within three years of getting the 
award; and 


to achieve ratings for “value for money” 
and “innovation” at least equal to those 
achieved in the 1990 evaluation and on the 
same basis. 


to support at least 60 projects in the first 
full year of the scheme and 150 projects 
annually by the third year which lead to a 
significant technical advance; 


to stimulate at least £25 million per year 
additional investment in research and 
development by SMEs by the third year; 


at least 80 new products to be marketed 
successfully by firms (grant holders and 
other firms) within two years of the end of 
the initial three year scheme; 


to generate additional profits in the 
companies supported, which when 
aggregated over the four years following 
the end of the project, amount to three 
times the grant; 


70% of projects to achieve most or all of 
their specified technical objectives; and 


to streamline administrative procedures so 
that at least 75% of offers are made within 
four months of the date of the initial 
application. 
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Overall and Specific Objectives by Scheme 


Scheme The Scheme’s Overall Objectives The Department’s Specific Primary 
Objectives and Non-Financial Targets 


RIN To improve the performance of small To encourage: 
businesses in qualifying areas through the 


improved efficiency and profitability; 
introduction of successful new products and ! Pot y 


processes leading to the increase or e increased turnover, 
safeguarding of employment. e — employment growth/safeguarding jobs; 
and 


e lessening the risk of stagnation or decline. 


Biotechnology Means Business To ensure that United Kingdom industry isina to establish a database of 2000 companies 
(phases 1 and 2) position to maintain its lead in the increasingly to be targeted in the sectors identified: 
competitive field of biotechnology by facilitating 25% to have attended an organised event 
the efficient dissemination of information and or requested specific information and 10% 
provide support for effective technology to have acknowledged that biotechnology 
transfer to industry; to provide appropriate has influenced investment decisions; 


techno-economic and market information to 
SMEs; to provide independent fora for 
information exchange , steerage and discussion 
on biotechnology; to improve industrial 
competitiveness through promotion of best 
practice; and reduce industrial risk by e to publish a revised “Plain Man’s Guide”; 


promoting demonstrator and feasibility projects. e to publish four newsletters per year; 


to achieve at least 60% satisfaction rate 
from those attending seminars, 
workshops, exhibitions and from those 
requesting publications; 


e — to support five demonstrator projects, 
each involving at least two partners (one 
industrial) in the live demonstration of 
prototype equipment and processes in 
biotechnology and promotion of quality 
assurance through adoption of best 
practice; 


e to commission and publish five 
state-of-the-art reports for dissemination 
to industry, particularly SMEs, to spread 
awareness of the benefits offered by 
biotechnology; 


e —_ to provide for 19 projects involving the 
employment by companies of experts to 
undertake strategic studies which will 
provide techno-commercial information 
for informed decisions about future 
investments; and 


e to provide for 21 projects involving the 
employment by companies of external 
expertise to undertake technical feasibility 
work. 

continued 
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Overall and Specific Objectives by Scheme 


Scheme The Scheme’s Overall Objectives The Department's Specific Primary 
Objectives and Non-Financial Targets 


Manufacturing Intelligence To increase manufacturing industry’s e to distribute in the first year to 3300 
knowledge of the advantages of new medium-sized companies (200-499 
technology, thus reducing fear of it: to employees), and other organisations 
encourage strategic decision makers and associated with the scheme, a high quality 
operational managers to consider the information pack; 


opportunities knowledge based systems 
technology offers for improving the 
competitiveness of their companies. 


12 issues of 2000 copies of a newsletter 
will be published over the life of the 
scheme and distributed to manufacturing 
companies and associated organisations; 
the projected readership is 6000 
individuals; 


e to demonstrate operational applications at 
first hand to companies considering 
implementing a knowledge based system 
solution in a particular area. It is 
anticipated that there will be 300 visitors 
to the host companies in years 2 and 3 of 
the scheme; 


e _ to set up six technology transfer clubs 
whose aim is to spread awareness of 
Manufacturing Intelligence among their 
members and provide a forum for mutual 
help and provide a mechanism for 
members to gain practical experience in 
building knowledge based systems. The 
aim is to have 300 companies join these 
clubs of whom 50 will implement systems 
by way of joint venture activities. In 
addition, the clubs will hold 12 seminars; 


e tohave up to 12 application oriented 
collaborative projects exploring novel 
applications in manufacturing established 
by the end of the second year in which 50 
companies will participate. The results to 
be disseminated via the newsletter; 


e of the 650 target companies, it is expected 
that some 150 will build or be building 
operational systems by the end of the 
scheme. Reportage of about 50 of these 
systems will demonstrate the 
opportunities afforded from using the 
technology and provide the basis for its 
spread to other medium-sized companies; 

continued 
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Overall and Specific Objectives by Scheme 


The Scheme’s Overall Objectives 


To contribute to an improvement in the 
competitiveness of British business by 
encouraging its target audience to address key 
management issues vital to success. To 
improve awareness of the need to manage 


change effectively, to promote understanding of | 


aspects of management practice important to 
competitive success and to encourage action to 
improve performance. 
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Objectives and Non-Financial Targets 


the annual competition for the 
Manufacturing Intelligence Award will 
continue. The competition encourages 
companies with successful systems to 
declare these, so providing evidence of 
real business benefits and lessons learned. 
The target is to have a minimum of 30 
entrants per year of a very high standard 
to publish the Executive Summaries. 
4000 copies of the Executive Summaries 
will be distributed each year ona 
complimentary basis and on sale through 
HMSO; and 


the scheme will aim to have 80% of those 
completing exit questionnaires for the 
Department of Trade and Industry events 
rate them as beneficial. 


at least 75% of those attending seminars 
and workshops will rate them as good or 
very good in respect of presentation and 
that at least 70% will rate them good or 
very good in respect of content; 


60% of those attending seminars and 
workshops should take action following 
their attendance, of whom 70% shall 
consider themselves to have been 
materially influenced by the event, 


70% of attendees to demonstrations 
should consider the content good or very 
good and that 60% should subsequently 
take action; 


to visit at least 30 sites with the strategy 
roadshow with total attendance of at least 
6000 people (1993-94 target); 


to hold at least 80 other seminars and 
workshops with total attendance of at least 
4500 people (1993-94 target); 


to organise at least 400 visits to 
demonstration firms involving at least 
3000 attendees (1993-94 target); 


to dispatch at least 400000 booklets and 
loan 30000 videos to interested 
companies (1993-94 target); 
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Overall and Specific Objectives by Scheme 


Scheme The Scheme’s Overall Objectives 


Teaching Company Scheme To facilitate the transfer of technology and the 
spread of technical and management skills and 
encourage industrial investment in training, 
research and development; to provide industry 
based training, supervised jointly by academic 
and industrial staff, for young graduates 
intending to pursue a career in industry; and to 
enhance the level of stimulating collaborative 
research and development projects and forging 
lasting partnerships between academia and 
business. 


Source: the Department's Statements of Objectives for each scheme. 


The Department's Specific Primary 
Objectives and Non-Financial Targets 


to create a new private sector quality 
award, building on the achievements of 
the British Quality awards; and 


to promote marketing and new production 
introduction as full topics within the 
programme. 


targets set for the number of programmes, 
new companies and new academic 
institution departments supported by the 
scheme overall in the years 1991-92 to 
1995-96. The Department support the 
majority of these programmes; and 


60% of all live programmes to have 
industrial partners whose total number of 
employees do not exceed 500 in the 
company or ultimate group. 


Appendix 3 shows by scheme the overall scheme objectives and the Department's specific primary objectives and non-financial targets, where these 


are specified. 
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Appendix 4 


Appraisal criteria and procedures for 
technology development schemes 


The appraisal criteria and procedures for the four technology 
development schemes examined by the National Audit Office are 
described below. 


LINK 


The LINK scheme comprises a number of technology specific 
programmes each with its own management committee and each 
supporting a number of individual collaborative projects. Each 
management committee uses a project co-ordinator who assists in 
the formation of a collaborative group to undertake LINK projects 
although the extent to which this support is provided varies from 
project to project and between programmes. The appraisal and 
approval arrangements for each project proposal are summarised in 
Figure 24 opposite. Projects cannot proceed until both the 
Department and the relevant research council have completed their 
appraisals and given their approval. 


Appraisal of project proposals 


Technical Approval 


The management committees are responsible for the technical 
appraisal of the project proposals. The appraisal concentrates on 
the technical viability of the project and whether it falls within the 
remit of the programme. It also assesses the commercial viability of 
the project although the depth of appraisal can vary according to the 
particular management committee involved and whether the 
proposal is at outline or full stage. 


Financial and Contractual Appraisal 


If the full proposal receives the approval of the management 
committee it is then appraised on financial grounds both by the 
Department and relevant research council simultaneously. The 
Department’s appraisal concentrates on the industrial partner and is 
carried out against the following criteria: additionality; project costs 
and their allocation between the partners; the project timescale; the 
contractual collaborative arrangements, including the roles of the 
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Figure 24: Arrangements for financial and technical appraisal and approval of LINK projects 


Participants come together. 
Develop outline proposal with help of 


programme coordinator 


outline proposal 


Programme Management Committee 
consider outline proposal 


approval 


Participants develop full proposal 


full proposal 


Programme Management Committee 
consider full proposal 


technical approval 


Research Council considers funding of 
academic partner 


Department considers funding of 
industrial partner 


financial approval financial approval 


Department seeks information and 
evidence of collaborative agreement from 
industrial partner 


Research Council seeks information and 
evidence of collaborative agreement from 
academic partner 


Liaison between Department 
and Research Council 


Department gives final approval Research Council gives final approval 


liaison 


Source: The Office of Science and Technology and the National Audit Office. 


Figure 24 shows the lengthy appraisal and approval arrangements for individual collaborative projects under the LINK scheme. 
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individual collaborators, with particular reference to SMEs and 
managerial arrangements; intellectual property rights; and the 
financial viability both of the project and the individual companies. 


SMART, SPUR and RIN 


The quality of project proposals under the SMART, SPUR and RIN 
schemes are appraised by regional offices on the basis of formal 
applications submitted by the companies concerned. The specific 
criteria applied under each scheme are set out in Table 9 below. In 
all cases the Department also appraise the companies’ assessment 
of their eligible project costs. 


Table 9: Appraisal Criteria by Scheme 


Appraisal Criteria Scheme 
SMART SPUR RIN 

Innovativeness (1) ° ° ° 
Additionality ° ° n 
Management Ability and Company Expertise ° ° : 
Commercial Viability of Project ° ° ° 
Technical Viability : e e 
Financial Viability of Project ° e = 
Financial Viability of Company ° ° - 
Intellectual Property Rights ° 6 = 
Business Plans 6 e ° 
Marketing Proposals e e : 
Improved Efficiency and Profitability - - ° 
Increased Turnover : S ° 
Employment Growth z - e 
Reduced Risk of Stagnation - - ° 


Source: the National Audit Office. 


Note: (1) The innovativeness test is less stringent for RIN than the other two schemes. 
For SMART, the project must be innovative for the United Kingdom as a whole 
whilst SPUR applicants must demonstrate a significant technological advance. 
For RIN, applicants need only demonstrate that the project is innovative for the 
company. 


Table 9 shows the specific appraisal criteria applied under the SMART, SPUR and RIN 
technology development schemes. 


The SMART scheme is a competitive scheme - applications must be 
submitted to the Department by a set date each year and only a 
fixed number of grants are awarded each year. To identify the 
winning applicants a formal marking system is used to appraise the 
quality of the project proposals using the specific criteria set out in 
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Table 9 above. Project proposals under the SPUR and RIN schemes 
are not formally marked and ranked in this way - each proposal is 
assessed on its own merits against the related appraisal criteria. 


Information required for appraisal of grant 
applicants 


7 The information required by the Department to assess proposals 
under these four schemes includes: 


e proof of eligibility (for instance, company size by number of 
employees); 


e detailed project data, including: 
- duration and description of project; 
- business plans and cash flows; 


- contractual and financial details of collaborative 
arrangements (LINK); and 


- project costs by specified categories. 
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Reports by the 
Comptroller and 


Auditor General 
Session 1994-95 


The Comptroller and Auditor General has to date, in Session 
1994-95, presented to the House of Commons the following reports 
under Section 9 of the National Audit Act, 1983: 


Treasury Management in National Health Service Trusts 


in Briglaridhh, reer vee cteeocstan eee tas ee cones rete ee ornare nearer HG 
The Financial Health of Higher Education Institutions 

Ty Egham ee ee ee nas cues tee esac eset ncenec sere eee HGsi3 
The Management of Intellectual Property in The Ministry of 

Agriculture, Fisheries and POOds.. sss 5 -c.t tee ee HC 15 
General Practitioner Fundholding in England..........................00. HG 
Overseas Development Administration: Management of 

Programme Ald sec. \ccscavsessnre stecceenidea tre nee acter nmaere HC 68 
Department for Education: Management of Office Space............. HG 
Crown Office and the Procurator Fiscal Service: Scottish 

Courts Administration Resources in Sheriff Courts. ............... HG119 
Resource Accounting and Budgeting in Government................. HGsi23 
Department of Transport: Sale of DVOLT (2. ern escsene en c2eeceeeae HC 128 
Sale of Forward Catering Services Limited ...............::.::ccseeseeeees HC 150 
Managing to be Independent: Management and Financial 

Control at Colleges in the Further Education Sector.............. HG 
Severance Payments to Senior Staff in the Publicly Funded 

Education: S@CtO¥ wsiccccchccowieccueen eseee -deceeree no aeseeeeseonce neuen: HC 202 


Second Sale of the Government's Debt in British 
Telecommunications and Privatised Electricity 


COM Panes ia te. eins dlee seated Sececadacsees aur eens ee Oeeereecseaee, ace HC 184 
Entry into the United: King domiss: 2. sccetey sere teeseeeeceenaaes HC 204 
HM Customs and Excise: Writing Off VAT Arrears ...............:..... HC 209 
IT Security in Government Departments .. 7. circ cescstenecenaeces HG 231 
Inland Revenue: Market Testing the Information 

Technology Of1Ce eter tree enn rear tere etree HC 245 
Ministry of Defence: The Risk of Fraud in Defence 

PrOCUTETNUCTIO ire ieiielerseree ere eee ad etme Eamte EONIGR eSaRES se HO258 
Contracting for Acute Health Care in England ..................00 HC 261 
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National Rivers Authority: River Pollution from Farms 


Vi 4 VACA D + cba soncecy peritlaly Ree ee oN an iy aR eee ee SAREE HC235 
Interim Report: PSA Services - The Sale of PSA Projects............ HC 306 
Commission for the New Towns: Disposal of Land and 

RON BY Ooi SAULT Sy CC ie allen oa Ron tr ee ee HC 308 
Sale of County Hall (Riverside Building) to Shirayama 

SEO Kroc Hu OUAT) cha Va EET LOC iremecmn pete ee ocak eaten ecseicans' canto’ HC 314 
Administration of Retirement Pensions ..................ccccccceeeseeeeeees HC 360 
Department of Employment: Financial Controls in 

Training and Enterprise Councils in England........................ He 364 
National Health Service Outpatient Services in England 

ELC Clie SIMONE eran cote te nese nero me ey Gods csh onthe chit eis eibon HGso9 
Department of Social Security: Purchase of Postal 

UC CUE Cle SOL VICCSU et ar, faites faa eee ion taeeis y ceaceaacbeseen teed HC 362 
Medical Research Council: Sale of the Mount Vernon Site, 

WANS Cea eer eradicate coy gee nent oa nctinitgia tle w scan focand sate cc HC 363 
Ministry of Defence: Management of the Capital Works 

oO SRO UITING Re eee terrane ee oe ey mean Seok eetee, Rane Be saan sansa HC 417 
Ministry of Defence: Major Projects Report 1994...................... HC 436 
Protecting and Presenting Scotland’s Heritage Properties.......... HC 430 
A Review of the Financial Controls over Indirectly 

Funded Operations of the Metropolitan Police Service.......... HC 462 
London Docklands Development Corporation: 

pa TERE AC TIO CaM eee ert cere Oe cat eee koe tarry: suitcs cases vss HC 468 
The Banking Service provided by The Office of 

PAID 3 Valin Leta GO Mera are peer. ascension tc asst esa sens Acct sawed avers edocs HG'513 
Evaluating the Applications to Run the National Lottery............ HC 569 
Property Holdings: Acquisition of Freehold Property ................. HC 570 
Ministry of Defence: Procurement Lessons for the New 

GONG a OU eE EI Lee meee ee eed ca ncnda sscesinesiesscseonteleseveteskes HC 692 
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